Speech, Hearing and Language: work in progress Volume 14 A CHOICE THEORY METHOD FOR EVALUATING AUDIOVISUAL PHONEME RECOGNITION

This article describes a mathematical method, based on Choice Theory (e.g., Luce, 1963), that can be used to predict audiovisual phoneme confusion matrices from unimodal audio and visual data. The predictions made from this method can be compared to obtained levels of audiovisual processing, for the purpose of identifying individuals whose audiovisual integration processes are not efficient. A reanalysis of Grant et al.'s (1998) audiovisual consonant confusion data is presented to evaluate this method. The results demonstrate that this method is effective at predicting audiovisual phoneme recognition responses, and suggests that Grant et. al's. subjects were highly efficient at integrating audiovisual information. Matlab code used in these analyses is available at http://www.phon.ucl.ac.uk/home/paul/CT/home.htm. Introduction Several methods have been developed to predict audiovisual phoneme confusion matrices based on phoneme confusion data collected under unimodal audio and visual stimulus conditions (Blamey, 1989; Braida, 1991; Massaro, 1987). This article describes a new method, based on Choice Theory, which serves the same purpose. Compared to other existing methods, the present method offers at least two advantages. First, this method is designed to estimate audiovisual phoneme recognition under optimal-processing conditions (see Braida, 1991; Grant et al., 1998); it estimates the highest level of audiovisual consonant recognition that is possible given the phonetic information available separately through the auditory and visual modalities. Predictions based on optimal processing assumptions are useful, because they can be compared to obtained levels of performance to help identify individual patients whose cognitive/perceptual processes (e.g., processes that integrate the phonetic information from each modality and map the phonetic information onto long-term memory representations for language) are not making efficient use of the available phonetic information (Grant et al., 1998; Grant & Seitz, 1998). Second, this method is less complex mathematically than the only other proposed method for estimating audiovisual performance under optimal-processing conditions, Braida's pre-labeling model (1991). Braida's pre-labeling model is based on a multidimensional extension of Signal Detection Theory (e.g., Durlach & Braida, 1969; Green & Swets, 1966; Macmillan et al., 1988). It requires fitting the consonants to locations within a multidimensional audiovisual perceptual space, calculating response regions for each consonant within this space, and integrating a multidimensional Gaussian probability function for each consonant over each of these response regions. In contrast, the Choice Theory method used here does not require the consonants to be represented within a multidimensional space (although it is possible to represent Choice Theory coefficients 1 Matlab code used in these analyses is available at http://www.phon.ucl.ac.uk/home/paul/CT/home.htm.